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METHOD FOR REDUCING MOTION BLUR IN A DIGITAL IMAGE 

Technical Field of the Invention 

5 The present invention relates to reducing motion blur in a digital image 

and, more particularly, to analyzing image data representative of a digital 
image to determine the amount and direction of motion blur and processing 
the image data to reduce the motion blur. 

10 Background of the Invention 

Digital still cameras generate image data representative of an image 
of an object. The process of generating image data representative of an 
image of an object is often referred to simply as "imaging" or "capturing" the 

15 object. The image data is processed and output to a device that displays a 
replicated image of the object. For example, the replicated image of the 
object may be displayed on a video monitor or printed by a printer. 

The digital camera focuses the image of the object onto a two- 
dimensional array of photodetecting elements. The photodetecting elements 

20 are relatively small and each one generates image data representative of a 
very small portion of the image of the object. For example, the two- 
dimensional array may have several million photodetecting elements that 
each generate image data representative of a small portion of the image of 
the object. The image data generated by the individual photodetecting 

25 elements is processed to recreate the image of the object. One type of 
photodetecting element is a charge-coupled device that outputs a voltage 
that is proportional to the amount of light it receives over a preselected 
period. 

In order to improve the image generated by a digital camera, the 
30 density of photodetecting elements on the two-dimensional array is 

increased. The increased density of photodetecting elements increases the 
number of photodetecting elements that image an object, which in turn 
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improves the quality of the image by reducing spaces between imaged 
portions of the object. Another method of improving the image generated by 
a digital camera, especially in low light conditions, is by using a long period to 
generate the image data. This long period of image generation is achieved 
5 by having the photodetecting elements detect light for an extended period. 

One problem with photography, including digital photography, is that 
the image generated by the camera will be blurred if the camera moves as 
the photodetecting elements generate image data. For example, under ideal 
conditions of no movement in the digital camera relative to the object being 

10 imaged, each photodetecting element generates image data representative 
of a particular portion of the image of the object. If, however, the camera is 
moved as the image data is generated, the individual portions of the image 
of the object will be imaged by several photodetecting elements. 
Accordingly, each photodetecting element images several different portions 

15 of the image of the object, which causes the replicated image of the object to 
be blurred. This type of blur is referred to as motion blur. 

The motion blur problem is exacerbated as the above-described 
imaging period is extended. The extended imaging period results in a higher 
probability that the camera will be moved during the imaging period, which 

20 results in a higher probability that motion blur will occur in the replicated 

image. Accordingly, the benefit of an extended imaging period may be offset 
by a higher probability of generating a blurred image. The problem of motion 
blur is further exacerbated by the use of more photodetecting elements to 
generate an image. The photodetecting elements will be smaller and receive 

25 less light. Accordingly, the exposure time of the camera must be extended 
in order for the smaller photodetecting elements to receive enough light to 
generate accurate image data. 

Therefore, a need exists for a method and device for detecting and 
reducing motion blur in a digital image. 
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Summar y of the Invention 

The present invention is directed toward minimizing motion blur in a 
replicated image of an object. The image of the object is represented in the 
5 form of image data. The image data may, as an example, be generated by 
photographing the object using a digital camera, which generates image data 
representative of the object. The image data may be a plurality of values 
corresponding to the intensity of light generated at small portions or pixel 
locations within the image of the object. 

1 0 The image data is analyzed to determine if the magnitude of the 

amplitude at a specific spatial frequency in one direction is less than the 
magnitude of the amplitude at the specific spatial frequency in other 
directions, which is indicative of motion blur. This may be generalized to 
multiple specific spatial frequencies by comparing the amplitude at each 

1 5 specific spatial frequency for the various directions. 

The magnitude of the amplitude of a specific spatial frequency is 
proportional to a figure of merit. A first figure of merit is calculated in a first 
direction and a second figure of merit is calculated in a second direction, 
wherein the second direction is substantially perpendicular to the first 

20 direction. If motion blur exists with respect to either the first direction or the 
second direction, the figure of merit in one direction will be smaller than the 
figure of merit in the other perpendicular direction. The magnitude of the 
ratio of the first figure of merit to the second figure of merit is compared to a 
preselected value to determine if the value of the first figure of merit is 

25 significantly different from the value of the second figure of merit. If the 
magnitude of the ratio is greater than the preselected value, motion blur 
exists. The direction of the motion blur is in the direction corresponding to 
the lesser of the two figures of merit values. 

Motion blur is minimized by increasing the magnitude of the amplitude 

30 of the signals in the appropriate set of spatial frequencies of the image in the 
direction of the motion blur. The appropriate set of spatial frequencies are 
dependent on the image characteristics. In one embodiment, the spatial 
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representation of image data may be improved in the direction of the motion 
blur by applying a deconvolution or modified deconvolution correction to the 
frequency domain representation of the image data. The frequency domain 
representation of the image data may then be converted back to the spatial 
5 representation of the image data. This conversion may, as an example, be 
accomplished by application of kernel operators in the spatial domain during 
a demosaicing operation. These operations may be accomplished by factory 
preselected correction factors, by image dependent correction factors, or by 
manually chosen correction factors. 

10 

Brief Description of the Drawings 

Fig. 1 is a schematic illustration of a digital camera imaging an object. 
Fig, 2 is a front, enlarged view of a two-dimensional photosensor array 
15 used within the digital camera of Fig. 1 . 

Fig. 3 is a color filter array using the Bayer pattern. 
Figs. 4A and 4B are a flowchart depicting a method of detecting and 
minimizing motion blur. 

Fig. 5A is a graph representative of photodetecting element outputs 
20 detecting a single point of light without the presence of a blur filter in the 
camera. 

Fig. 5B is a graph representative of photodetecting element outputs of 
the point of light of Fig. 5A with the addition of a blur filter 

Fig. 5C is a graph representative of photodetecting element outputs of 
25 the point of light of Fig. 5A with the addition of a blur filter and motion blur 

Fig. 6A is a graph showing the discrete spatial frequency 
representation of the graph of Fig. 5A. 

Fig. 6B is a graph showing the discrete spatial frequency 
representation of the graph of Fig. 5B. 
30 Fig. 6B is a graph showing the discrete spatial frequency 

representation of the graph of Fig. 5C. 

Fig. 7 is an enlarged view of photodetecting elements located on the 
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surface of the two-dimensional photosensor array of Fig. 2. 

Fig. 8 is a graph showing the representation of Fig. 5C with nriotion 
blur minimized. 

5 Detailed Description of the Invention 

A method and apparatus for detecting and reducing motion blur in an 
image is generally described below followed by a more detailed description. 
Figs. 1 through 8. in general, illustrate a method for detecting motion 

10 blur in a digital image. The method may comprise providing image data 

representative of the digital image. The image data is analyzed to calculate 
a first figure of merit of the digital image in a first direction (X) and a second 
figure of merit of the digital image in a second direction (Y), wherein the first 
and the second directions are substantially orthogonal. A first ratio of the 

15 first figure of merit to the second figure of merit is calculated, wherein the 
ratio is the greater of the first or second figure of merit is divided by the 
lesser of the first or second figure of merit. The first ratio is compared to a 
preselected value, wherein motion blur exists in the digital image if the first 
ratio is greater than the preselected value. 

20 Figs. 1 through 8 also, in general, illustrate a method for reducing 

motion blur in an image. The method may comprise providing image data 
representative of the image. The image data is analyzed to detect the 
presence of motion blur in the image. The image data is further analyzed to 
detect the direction of motion blur in the digital image. The image data is 

25 then processed to increase edge acuity the image in the direction of the 
motion blur. 

Figs. 1 through 8 also, in general, illustrate an apparatus for detecting 
motion blur in an image. The apparatus may comprise a computer and a 
computer-readable medium operatively associated with the computer, 
30 wherein the computer-readable medium contains instructions for controlling 
the computer to detecting motion blur in an image by the method described 
below. The apparatus receives image data representative of the image. A 
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first figure of merit of the image data is calculated in a first direction (X). A 
second figure of merit of the image data is calculated in a second direction 
(Y). wherein the first and the second directions (X, Y) are substantially 
orthogonal. A first ratio of the first figure of merit to the second figure of merit 
5 is calculated. The first ratio is the greater of the first or the second figure of 
merit divided by the lesser of the first or the second figure of merit. The first 
ratio is compared to a preselected value, wherein motion blur exists in the 
digital image if the first ratio is greater than the preselected value. The 
direction of motion blur corresponds to the lesser of the figures of merit. 
10 Having generally described the digital camera 100 and a method for 

reducing blur in a digital image, they will now be described in greater detail. 
The following description focuses on the operation of the camera 100 
followed by detection and reduction of motion blur in images generated by 
the camera 100. 

15 A schematic illustration of a digital camera 100 generating image data 

representative of an object 1 10 is illustrated in Fig. 1 . The process of 
generating image data representative of an object is sometimes referred to 
simply as imaging or capturing the object. An x-direction X and a z-direction 
Z are used for reference purposes with regard to the camera 100 and the 

20 object 110 shown in Fig. 1 . The x-direction X is described in greater detail 
with reference to a positive x-direction XP and a negative x-direction XN. 
Likewise, the z-direction Z is described in greater detail with reference to a 
positive z-direction ZP and a negative z-direction ZN A y-direction, not 
shown in Fig. 1. extends perpendicular to both the x-direction X and the z- 

25 direction Z and is described below. 

The camera 100 may have a housing 120 with an aperture 122 
formed therein. A lens 126 or a plurality of lenses may be located within or 
adjacent the aperture 122 and may serve to focus an image of the object 110 
onto components located within the camera as is described below. The lens 

30 126 may, as a non-limiting example, have a focal length of approximately 
seven millimeters. A two-dimensional photosensor array 130 and a 
processor 132 may also be located within the housing 120. As illustrated in 
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Fig. 1 . the image of the object 1 10 is focused onto the two-dimensional 
photosensor array 130 by the lens 126. It should be noted that the view of 
the two-dimensional photosensor array 130 illustrated in Fig. 1 is a side view. 
The processor 132 may be electrically connected to the two-dimensional 
5 photosensor array 130 and may serve to process data generated by the two- 
dimensional photosensor array 130 as is described in greater detail below. 

A front, enlarged view of the two-dimensional photosensor array 130 
is illustrated in Fig. 2. The two-dimensional photosensor array 130 may have 
a surface 136 to which a plurality of photodetecting elements 138 are 
10 mounted. It should be noted that the photodetecting elements illustrated in 
Fig. 2 have been greatly enlarged for illustration purposes. The 
photodetecting elements 138 may be arranged so as to form a plurality of 
O rows 140 and columns 142. Each of the photdetecting elements 138 may be 

g adapted to generate image data representative of the intensity of a specific 

f] 15 frequency band of light. For example, with additional reference to Fig. 3, a 
yj color filter array 143 may be attached to the surface 136 of the two- 

Si dimensional photosensor array 130 to filter the light before the light reaches 

1^ the photodetecting elements 138. Accordingly, each photodetecting element 

m 138 may receive light having a specific and narrow band of wavelengths. 

20 The color filter array 143 shown in Fig. 3 illustrates the Bayer patter, wherein 
Q R, G, and B represent color filters that allow specific bands of red, green and 

blue light to pass to the photodetecting elements 138. Accordingly, each of 
the red, green, and blue filters shown in Fig. 3 are associated with single 
photodetecting elements 138. It should be noted that the Bayer pattern has 
25 twice as many green filters as red and blue. The increased number of green 
filters compensates for human vision that is more sensitive to green light that 
red or blue light. It should also be noted that the color filter array 143 may 
have filters arranged in patterns other than the Bayer patter shown in Fig. 3. 
The photodetecting elements 138 may be charge-coupled devices 
30 that develop a charge that is proportional to the amount of light they receive 
during a preselected period. The preselected period is dependent on the 
intensity of light to which the object 110, Fig. 1 , is exposed and, thus, the 
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intensity of light 1 50 that reflects from the object 110. For example, when an 
image is generated in a relatively dark environment, the preselected period is 
increased to enhance the generated image. Likewise, when an image is 
generated in a relatively bright environment, the preselected period is 
5 decreased. 

Referring to Figs. 1 and 2, in the embodiment of the camera 100 
described herein, the lens 126 causes a blur of the image of the object 110 
onto the photodetecting elements 138. The blur in the embodiment 
described herein is equal to approximately two photodetecting elements 138 
10 and serves to prevent aliasing of the image during processing. 

Referring again to Fig. 1, as described above, the processor 132 may 
be electrically connected to the two-dimensional photosensor array 130. The 
p processor 132 receives and processes image data generated by the two- 

^ dimensional photosensor array 130. For example, the processor 132 

fff 15 processes information generated by the individual photodetecting elements 
y 138 to "demosaic" the image and to properly combine the colors as 

%l described in greater detail below. Some examples of the functions 

^ performed by the processor 132 are described in the United States patent 

m application, serial number 09/177,729 for IMAGE DEMOSAICING AND 

J 20 ENHANCEMENT SYSTEM of Taubman, filed on October 23, 1 998, which is 
O hereby incorporated by reference for all that is disclosed therein. It should 

^ be noted that the processor 132 described herein has the additional ability to 

detect and minimize motion blur in an image. 

The processor 132 may also be electrically connected to a peripheral 
25 viewing device, not shown, such as a computer monitor or a printer that 

serves to display and/or process the image data generated by the camera 
100. The processor 132 may also be electrically connected to a peripheral 
computer, not shown, that stores and/or processes the image data 
generated by the two-dimensional photosensor array 130 and. thus, the 
30 camera 100. It should be noted that the processing techniques and methods 
described herein with reference to the processor 132 may be performed by 
the peripheral viewing device or computer. For example, image data 
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generated by the two-dimensional photosensor array 130 may be transmitted 
directly to a peripheral processor for processing. 

Having described the components of the camera 100 that are 
essential for the correction of motion blur, the operation of the camera 100 
5 will now be described followed by a description of the camera 100 correcting 
for motion blur. 

The camera 100 is typically held by a user and used to generate 
image data representative of an object, such as the object 110. The image 
data typically represents a still picture of the object 110, however, the image 

10 data may represent a moving picture, e.g., a motion picture, of the object 
110. Light 150 reflects from the object 110 and enters the housing 120 of 
the camera 100 via the aperture 122. The lens 126 then focuses an image 
of the object 110 onto the two-dimensional photosensor array 1 30. The two- 
dimensional photosensor array 130 generates image data representative of 

15 the image of the object 110, which is output to the processor 1 32 for 

processing. As described above, in the embodiment of the camera 100 
described herein, the lens 126 blurs the image focused onto the two- 
dimensional photosensor array 130 by two photodetecting elements 138, Fig. 
2. 

20 Fig. 2 illustrates an example of an image 1 56 of the object 110, Fig. 1 , 

focused onto the two-dimensional photosensor array 130. When the image 
156 is properly focused onto the two-dimensional photosensor array 130, an 
instruction is transmitted to the two-dimensional photosensor array 130, 
which causes the photodetecting elements 138 to generate image data. 

25 More specifically, the photodetecting elements 138 develop electrical 
charges that are dependent on the amount of light they receive over a 
preselected period that is reflected from the object 110. When the camera 
100, Fig. 1, is used in dark environments, the preselected period is relatively 
long and when the camera 100 is used in bright environments, the 

30 preselected period is relatively short. 

Referring to Figs. 2 and 3, the image data may be processed as a 
plurality of individual color planes. For example, the image data may have a 
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green plane, a red plane, and a blue plane. Alternatively, the green plane 
may be processed as two green planes because there are twice as many 
green photodetecting elements 138 as red or blue photodetecting elements 
138. The green plane consists of image data generated by photodetecting 
5 elements 138 having a green color filter of the color filter array 143 

associated with them. Likewise, the red and blue planes consist of image 
data generated by photodetecting elements 138 having red and blue color 
filters of the color filter array 143 respectively associated with them. These 
color planes allow for each color component of the composite image data to 

10 be processed individually. For example, individual colors may be scaled to 
increase or decrease their relative intensity. Likewise, image data from 
individual photodetecting elements 138 may be scaled accordingly. 

The color filter array 143 provides a reference for directions used 
herein, including the directions of motion blur. A y-direction Y extends 

15 perpendicular to the x-direction X that was described above. An a-direction 
A extends diagonally to the x-direction X and the y-direction Y. A b-direction 
B extends perpendicular to the a-direction A. The aforementioned directions 
provide non-limiting examples of blur motion and correction as will be 
described in greater detail below. 

20 Referring again to Fig. 1. under ideal circumstances, both the camera 

100 and the object 110 remain stationary relative to each other during the 
above-described preselected period in which the two-dimensional 
photosensor array 130 generates image data. Ideal circumstances, 
however, are rarely achieved and the camera 100 generally moves 

25 somewhat relative to the object 1 10 while image data is being generated. 

For example, a user typically is unable to maintain the camera 100 in a fixed 
position relative to the object 110 while image data is being generated. 

Motion blur is typically attributed to either rotational or translational 
movement of the camera 100 relative to the object 110 occurring as the 

30 object 1 10 is being imaged. The motion blur correction methods described 
herein are directed toward correcting motion blur caused by translational 
motion between the object 110 and the camera 100 as the object 100 is 
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being imaged. It should be noted, however, that motion blur caused by 
rotational motion may also be corrected by the methods described herein. 
For example, if the center of relative motion between the camera 100 and 
the object 1 10 is a significant distance from the image area, the motion blur 
5 associated with the image may be substantially similar to motion blur caused 
by translational motion. One test to determine if motion blur attributed to 
rotational motion can be corrected as though it is attributed to translational 
motion is my measuring the blur on the image closest to the center of 
rotation and furthest from the center of rotation. If the two blurs are 
10 substantially equivalent, the motion blur may be corrected as though it is the 
result of translational motion. In one embodiment of the methods described 
herein, the blurs at the edges of the image are measured to determine if they 

o are substantially equivalent, meaning that the blur is the result of 

2 translational motion. 

■ass* 

15 Having summarily described the operation of the camera 100, the 

W operation of the camera 100 will now be described generating image data 

ni that is blurred. In the example illustrated herein, the camera 100 has moved 

in the x-direction X as the image data was being generated. The movement 
Ul in the x-direction X is an amount that causes the image 156 to move a 

20 distance of two and one-half photodetecting elements 1 38, 
O The movement of the camera 1 00 relative to the object 1 1 0 as the 

two-dimensional photosensor array 130 generates image data results in the 
image of the object 110 being blurred. More specifically, when the image 
data is processed using conventional processing methods, the replicated 
25 image of the object 110 will be blurred. An example of this blurring is 
illustrated in Fig. 2. The image 156 of the object 110, Fig. 1, is shown 
focused onto the two-dimensional photosensor array 130 at the 
commencement of the generation of the image data. As the generation of 
the image data by the photodetecting elements 138 proceeds, the image 156 
30 focused onto the two-dimensional photosensor array 130 moves in the 

negative x-direction XN. By the time the photodetecting elements 1 38 have 
ceased generating image data, the image 156 has moved to a different 
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location on the two-dimensional photosensor array 130. The location of the 
image of the object 110. Fig. 1. at the termination of the generation of image 
data is referenced as the second image 158 and is illustrated by dashed 
lines of Fig. 2. In the example cited herein, the image 156 has moved a 
5 distance that is represented by two and one-half columns 142 in the negative 
x-direction XN during the generation of the image data. The distance that 
the image 156 moved is proportional to the amplitude of the motion blur. 
The direction that the image 156 moved is referred to as the direction of the 
motion blur. 

10 As described in greater detail below, image data is analyzed to 

determine the direction and amplitude of the blur. The amplitude is the 
amount of movement that occurred during the generation of the image data. 
The direction of motion blur as described herein is. for illustration purposes 
only, limited to the x-direction X, the y-direction Y, the a-direction A and the b- 

15 direction B. It should be noted that the motion blur detection and correction 
methods described herein are applicable to a plurality of amplitudes and 
directions. 

Referring again to Fig. 1 , the image data generated by the two- 
dimensional photosensor array 130 is transmitted to the processor 132 for 

20 processing. The processor 132 correlates the image data generated by each 
of the photodetecting elements 138, Fig. 2, with its associated color filter per 
the color filter array 143 of Fig. 3. For processing purposes, the image data 
may be sorted into the three color groups or planes described above, which 
are the red plane, the green plane, and the blue plane. Alternatively, the 

25 image data may be sorted into four groups if two green color planes are 
considered. 

The processor 132 may store a numeric value corresponding to the 
image data values, or simply image data, generated by each of the 
photodetecting elements 138. The image data generated by each of the 
30 photodetecting elements 138 is proportional to the intensity of light received 
by each of the photodetecting elements 138. The processor 132 may also 
store the location from where each value of the image data was generated on 
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the two-dimensional photosensor array 130. 

Having described the generation of image data, the detection and 
reduction of motion blur will now be described. The following procedure is 
outlined in the flowchart of Figs. 4A and 4B. 
5 Image data generated by the two-dimensional photosensor array 130 

is representative of the amplitude of light at a plurality of spatial locations 
within the area of the image. In the situation where the image data is 
representative of a color image, the image data is representative of the 
amplitude of light at a plurality of spatial locations for a plurality of color 

10 planes. For example, the image data may be represented in three color 
planes, a red plane, a green plane, and a blue plane. The image data of 
each color plane may be transformed to a frequency domain by application of 
a Fourier transform and is sometimes referred to as the "transformed" image 
data. The transformed image data represents an amplitude and a phase for 

1 5 each of a set of spatial frequencies. 

The transformed image data offers a perspective on the image data 
relating to the amplitude and direction of blur within the image. The 
transformed image data may be manipulated as described below to reduce 
motion blur. The image data may then be retransformed by way of an 

20 inverse Fourier transform to its original format for presentation to a user. The 
motion blur in the retransformed image data has been reduced by the 
manipulation. 

Motion blur reduces the amplitude of the transformed image data, 
especially in the higher spatial frequencies and always in the direction of the 

25 blur motion. When the image data is transformed into the frequency domain 
using a Fourier transform or the like, the magnitude of the amplitude of 
sinusoidal signals throughout the set of spatial frequencies in a direction 
perpendicular to the direction of the motion remains virtually unaffected. The 
same occurs with regard to directions that are close to perpendicular to the 

30 direction of the motion. 

Having summarized motion blur, its detection and minimization will 
now be described in greater detail. 
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An example of the spatial affects of motion blur are shown in the 
graphs of Figs. 5A through 5C. The graphs 5A through 5C are, for illustration 
purposes, based on a linear view of the two-dimensional photosensor array 
1 30 rather than a planar view of the two-dimensional photosensor array 1 30. 
5 The horizontal axes P in the graphs 5A through 5C represent spatial 

locations, and more particularly photodetecting element locations or pixel 
locations, from where image data was generated. Accordingly, the horizontal 
axes correspond to the locations of the photodetecting elements 138 of Fig. 
2. The vertical axes l(P) represent the intensity of light received at the 
10 photodetecting element locations, which are values or magnitudes of the 

image data generated by the photodetecting elements. The graphs of Figs. 
6A through 6C represent the magnitudes of the amplitudes of the spatial 
O frequencies of the graphs of Figs 5A through 5C respectively, 

"g The graph of Fig. 5A shows the ideal representation of image data l(P) 

^ 15 of a single point of light that would be generated by a two-dimensional 
hi photosensor array without a blur filter associated therewith. As shown in Fig. 

5A, a single photodetecting element, eight or spatial location eight, generates 
image data having a magnitude of eight. The graph of Fig. 5B is a 
Ln representation of image data l(P) of the single point of light of Fig. 5A that 

^» 20 would be generated by a two-dimensional photosensor array having a blur 
O filter of two pixels associated therewith. As shown in Figs. 5A and 5B, the 

blur filter causes the image data l(P) to have a reduced high spatial frequency 
content, which is evident by the discrete pulse of the graph of Fig. 5A versus 
the discrete triangular function of the graph of Fig. 5B. In the graph of Fig. 
25 58, the magnitude of the spatial location eight drops to an amplitude of four 
and the magnitudes of spatial locations seven and nine increase from zero to 
two. It should be noted that the graph of Fig. 58 is considered the "perfect" 
point of light as imaged by the camera with the two pixels of blur. As the 
number of photodetecting elements in a digital camera increases, this 
30 "perfect" image represented by the graph of Fig. 58 more closely resembles 
the point of light of Fig. 5A. Accordingly, the negative affects of the blur filter 
become insignificant. 
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It should be noted that many digital cameras have various types of blur 
filters, such as a blurry lens, a birefringent filter or lens, or other blur filter 
device. The blur filter is to prevent objectionable aliased signal artifacts in the 
image. Aliased signal artifacts are typically associated with large areas of 
5 moderately high spatial frequency repetitive patters, such as the weave of a 
shirt. 

The graph of Fig. 5C shows the amplitude of image data received by a 
camera having a blur filter associated therewith and undergoing motion 
equivalent to two and one-half pixels or photodetecting elements as image 

10 data is being generated. As shown in Fig. 5C, the energy from a single point 
of light (Fig. 5A) is spread over a total of five photodetecting elements due to 
the blur filter and the motion blur. In the non-limiting example of Fig. 5C. the 
amplitude of image data generated at the sixth and tenth spatial locations has 
increased to 0.6. The amplitude of image data generated at the seventh and 

1 5 ninth spatial locations have decreased of two. The amplitude of image data 
generated at the eighth spatial location has decreased to 2.8. As shown by 
the graph of Fig. 5C, the image data generated with respect to the spatial 
locations represents a very wide triangular function. This is indicative of a 
very low magnitude of high spatial frequency content of the replicated image, 

20 meaning that the replicated image is not sharp and is likely blurred. 

As will be described in greater detail below, motion blur in the 
replicated image is reduced by increasing the magnitude of the amplitude of 
the sinusoidal signals at specific spatial frequencies of the frequency domain 
transformed image in the direction of the motion blur. It should be noted that 

25 the sinusoidal signals are used herein for illustration purposes and that they 
are derived by way of a Fourier transform. It should also be noted that the 
inventive concepts described herein are applicable to other signals derived 
from other transform functions. 

Different sharpening kernels may be applied to the image data in the 

30 direction of the motion blur to increase the magnitude of the amplitude of the 
high spatial frequency content. In one non-limiting example, a Weiner 
correction algorithm is applied to the frequency domain transformed image 
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data in the direction of the motion blur to increase the magnitude of the 
amplitude of a specific set of spatial frequency content. It should be noted 
that other correction algorithms may be applied to the image data to reduce 
motion blur. 

5 Having summarily described the effects of motion blur and a method of 

minimizing the effects of motion blur, the detection of motion blur will now be 
described. 

As summarily described above, motion blur due to translational 
motion, unlike other blurs, only occurs in a single direction. For example, with 
10 reference to the motion blur illustrated by the images 156 and 158 of Fig. 2, 
the motion blur only occurs in the x-direction X. Other blurs, such as optical 
blur, typically occur in all directions. Accordingly, the magnitude of the 
Q amplitude of the high spatial frequency content of an image affected by 

motion blur will be low in the direction corresponding to the direction of the 
ff; 15 motion. The magnitude of the amplitude of the high spatial frequency content 
bj corresponding to directions perpendicular or substantially perpendicular to the 

^ direction of motion will be less. When applied in practice, the magnitude of 

the amplitude of the high spatial frequency content in the direction of the 
tn motion blur will be significantly lower than the magnitude of the amplitude of 

20 the same specific high spatial frequency content in a perpendicular direction. 
O The detection of motion blur may be accomplished by analyzing image 

data generated by photodetecting elements 138 located in a portion of the 
two-dimensional photosensor array 130. A user of a camera typically assures 
that the subject of the photograph is focused and substantially centered in the 
25 photograph. Accordingly, the center of the photograph is where the sharpest 
focused image is located. For this reason, in the examples described herein, 
image data generated by a central portion of the two-dimensional 
photosensor array 130 will be analyzed for motion blur. For example, image 
data generated by the middle one-ninth of the two-dimensional photosensor 
30 array 130 may be analyzed for motion blur. It should be understood, 

however, that any portion of the two-dimensional photosensor array 130 may 
be analyzed for motion blur. 
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For illustration purposes, an example of analyzing image data to 
determine the direction and magnitude of motion blur is provided. The 
analysis commences analyzing image data generated by the photodetecting 
elements 138 located in the vicinity of a corner 160 of the image 156. Fig. 7 
5 is an exploded view of the photodetecting elements 138 located in the corner 
160 of the two-dimensional photosensor array 130. Image data generated by 
a row 164 of photodetecting elements 138 extending in the x-direction X will 
be analyzed in the following example. Likewise, image data generated by a 
column 166 of photodetecting elements 138 extending in the y-direction Y will 
10 also be analyzed in the following example. The photodetecting elements 138 
in the row 164 are referenced as X(1).Y(1) through X(10),Y(1) and the 
photodetecting elements 138 in the column 166 are referenced as X(1),Y(1) 
□ through X(1),Y(10). It should be noted that for illustration purposes, only ten 

S photodetecting elements 138 extending in the x-direction X and the y- 

^ 15 direction Y are being analyzed. It is to be understood, however, that any 
yj number of photodetecting elements 138 in the x-direction X and the y- 

direction Y may be analyzed. It is also to be further understood that any 
^ number of photodetecting elements 138 extending in the a-direction A and 

5 the b-direction B may also be analyzed to detect motion blur in other 

'S' 20 directions. 

Q The first step in determining whether motion blur exists in the image 

data is to analyze the magnitude of the amplitude of the high spatial 
frequency content in orthogonal directions over an area of an image. The 
magnitude of the amplitude of the high spatial frequency content in the x- 
25 direction X and the y-direction Y is proportional to the figure of merit in the x- 
direction X and the y-direction Y respectively. The figure of merit provides a 
basis for determining the degree to which light/dark transitions occur in a 
specific direction in an image. A non-limiting example of a figure of merit in 
the x-direction X is referred to herein as F^ and is calculated as follows: 



30 



F" = Z Z m) - X(n - 1, m)| 
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wherein p is the number of photodetecting elements 138 in the x- 
direction X that are to be analyzed, q is the number of rows that are analyzed 
in the x-direction X, and X(n,m) is the value of the image data generated by 
the photodetecting element 138 at that location. Accordingly, n designates 
5 the column number of the photodetector array and m designates the row 

number. It should be noted that in the non-limiting example described herein, 
the figure of merit Fx may calculated in the center of the image where the 
sharpest focus occurs. It should also be noted that the figures of merit are 
calculated in a single color plane and not by adjacent photodetecting 
10 elements for the example of the digital camera described herein. This 

assures that the figure of merit measures motion blur and not color transitions 
in the image. Furthermore, by not analyzing adjacent photodetecting 
Q elements, the anti-aliasing blur filter will have little, if any, influence on the 

fQ figure of merit calculation. It should be noted that virtually any increment of 

15 pixel values may be used to calculate the figure of merit. A preferred 
yj embodiment will use photodetecting elements that are close together within 

one color plane, because this will emphasize the higher spatial frequencies. 
1^ The higher spatial frequencies are the most sensitive measures of motion 

yl blur. It should also be noted that the figures of merit may be calculated in any 

20 direction to measure motion blur in any direction. 
P As with the figure of merit in the x-direction X, the figure of merit in the 

y-direction Y is referred to as Fy and is calculated as follows: 

F^ = ZZ|Y(n,m)-Y(n,m-l)| 

25 

wherein q is the number of photodetecting elements 138 in the y- 
direction Y that are to be analyzed, p is the number of columns in the y- 
direction Y, and Y(n.m) is the value of the image data generated by the 
photodetecting element 138 at that location. It should be noted that in the 
30 non-limiting example described herein, the figure of merit Fy is calculated 

along the column 166. In one preferred embodiment, p is equal to q so that 
the figures of merit are calculated from a square section of the image. 
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As described above, the figures of merits described herein are non- 
limiting examples of figures of merit. Other variations of the figure of merit 
can use weighted multiple color planes. For example, the figures of merit 
may be calculated in all the color planes. The values from the different color 
5 planes can be weighted and summed togther. For example, each green 

plane may be weighted thirty percent, the red plane may be weighted thirty 
percent and the blue plane may be weighted ten percent. Likewise, the 
spatial frequencies may be weighted. It is also possible to combine weighted 
color planes with weighted spatial frequencies. In another variation, higher 

10 order numerical methods may be used to generate the slope estimates. 
These slope estimate may be used rather than the simple difference 
calculations described above. 

The figures of merit provide indications of the magnitudes of the 
amplitudes of spatial frequencies of the image in the specified directions. 

15 Referring briefly to Fig. 5A, the graph shows a steep transition between 

adjacent photodetecting elements for image data generated without motion 
blur and without a blur filter. More precisely, the single point of light has 
caused image data to be generated solely at the eighth spatial position. 
Accordingly, the figure of merit associated with the graph of Fig. 5A is high. 

20 The graph of Fig. 5B shows transitions between image data values that are 
not as steep as the graph of Fig. 5A. Accordingly, the figure of merit 
associated with the graph of Fig. 5B is less that the figure of merit associated 
with the graph of Fig. 5A. The graph of Fig. 5C shows image data generated 
in the presence of motion blur. The motion blur decreases the magnitudes of 

25 the transitions between image data values of adjacent photodetecting 
elements 138. Fig 2. Accordingly, the figure of merit of the image data 
associated with the graph of Fig. 5C is less than the figures of merit 
associated with the graphs of Fig. 5A and Fig. 5B. It should also be noted 
that the figure of merit as described herein is proportional to the magnitude of 

30 the slope of the graphs. By using the figure of merit equations described 

above, the figures of merit for the graphs of Figs. 5A is sixteen, the figure of 
merit of the graph of Fig. 5B is eight, and the figure of merit of the graph of 
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Fig. 5C is 5.6. 

As described above, motion blur due to translational motion, unlike 
other blurs, occurs only in one direction. Accordingly, if the figure of merit in 
the x-direction X, Fx. differs from the figure of merit in the y-direction Y, Fy, 
5 motion blur likely occurred during the generation of the image data. This 

difference in the figures of merit may be calculated by taking the ratio of Fx to 
Fy or Fy to Fx and comparing the ratio to a preselected value. As a non- 
limiting example, if the ratio is greater than 1.4, it can be assumed that motion 
blur occurred. It follows that the direction of the motion blur will be in the 

10 direction having the lower figure of merit because the magnitudes of the 

transitions in this direction are lower. It also follows that the value of the ratio 
determines the amount of motion blur. For example, a greater value of the 
ratio means that the image was blurred over a greater number of 
photodetecting elements 138, Fig. 2. The value of the ratio is used to 

15 determine the amount to which motion blur is to be minimized. It should be 
noted that the non-limiting methods described herein for determining the 
direction of motion blur determine the direction of motion blur closest to a 
preselected direction. For example, the methods described herein may 
determine the direction of motion blur that is closest to the x-direction X, the 

20 y-direction Y, the a-direction A or the b-direction B. Similar methods can be 
used to expand the possible number of directions in which motion blur may 
be analyzed. 

When a determination has been made that the image has been 
blurred or, more specifically, that the replicated image is blurred, the image 

25 data is processed to minimize the blur of the replicated image. Minimizing 
motion blur involves increasing the magnitude of the amplitude of sinusoidal 
signals at specific spatial frequencies of the frequency domain transformed 
image data in the direction of the blur. A non-limiting example may be 
achieved by amplifying image data values corresponding to higher spatial 

30 frequencies. For example, in the case where the motion blur occurs in the x- 
direction X, the image data generated by individual rows 140, Fig. 2, of 
photodetecting elements 138 are processed to increase the magnitude of the 
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amplitude of spatial frequency. Other scaling and processing techniques as 
described below may also be applied to the image data. 

It should be noted that the effects of motion blur occur to all the color 
planes equally. Therefore, image data from one color plane may be analyzed 
5 to determine the direction and amount of motion blur. The magnitude of the 
amplitude of sinusoidal signals at specific spatial frequencies of the frequency 
domain transformed image of the color planes may be increased equally in 
order to minimize the affects of motion blur on the whole composite image. It 
should be further noted that processing the image data to minimize the 
10 motion blur may be performed during the process of demosaicing the image 
data as disclosed in the United States patent application of Taubman, 
previously referenced. This may, as an example, involve scaling 
p demosaicing coefficients in the direction of the motion blur to increase the 

magnitude of the amplitude of sinusoidal signals at specific spatial 
Oi 15 frequencies of the frequency domain transformed image in the direction of 
yi the motion blur. Other techniques can be used to reduce motion blur, such 

^: as directional sharpening kernels, windowed directional sharpening kernels, 

3^ and directional deconvolution correction. 

The procedure described above determines the amount and direction 
20 of motion blur. It should be noted that the procedure described above is 
O applicable for detecting and minimizing motion blur along any orthogonal 

directions. For illustration purposes, an example of detecting and minimizing 
motion blur in additional directions diagonal to the x-direction X and the y- 
direction Y is provided. The diagonal directions are referenced in Fig. 3 as 
25 the a-direction A and the b-direction B, which are perpendicular to each other. 
In this situation, the figures of merit are calculated in the a-direction A and the 
b-direction B along with the ratio of the figures of merit as described above 
with reference to the x-direction X and the y-direction Y. At this point, two 
ratios exist, one for figures of merit in the x and y-directions X and Y and one 
30 for figures of merit in the a and b-directions A and B. The ratio having the 

greatest value indicates the orthogonal directions associated with the motion 
blur. The actual direction of the motion blur is determined by the lowest 
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figure of merit associated with the ratio having the greatest value. Motion blur 
in this direction is minimized in the manner described herein by increasing the 
magnitude of the amplitude of the magnitude of sinusoidal signals at specific 
spatial frequencies of the frequency domain transformed image in this 
5 direction. 

Having described the method for determining whether motion blur is 
present in an image and a method of minimizing the motion blur, a detailed 
example of reducing motion blur is now provided. 

The following example is based on the imaged point of light of Figs. 5A 
10 through 5C. As described above, the point of light without the use of a blur 
filter appears solely at the eighth spatial position and generates image data 
having a magnitude of eight. The point of light imaged by a camera having a 
p two-pixel blur filter and undergoing motion blur of two and one half pixels 

^ generates image data as described above with reference to the graph of Fig. 

yi 15 5C. The values of spatial frequency of the graph of Fig. 5C are significantly 
yj less that the values of spatial frequency of the graph of Fig. 5A. For 

~ ; reference purposes, the values of the spatial frequencies of the graphs of 

- Figs. 5A through 5C are shown in the bar graphs of Figs. 6A through 6C. It 

m should be noted that the graphs of Figs. 6A through 6C do not include phase 

■r 20 information. As shown by the graph of Fig. 6A, the single point of light of Fig. 
O 5A has a normalized amplitude of one across the frequency spectrum, similar 

to an impulse function. The graph of Fig. 7B shows the decreased spatial 
frequency values due to the blur filter added to the camera. Fig. 7C shows 
that the values of spatial frequency are even more decreased by motion blur. 
25 The values of the spatial frequencies due to the blur filter and motion blur are 
detailed in Table 1 . It should be noted that in the spatial frequency eight is 
the highest and spatial frequencies zero and fifteen are the lowest. 
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Table 1 



Ideal Blur Filter Motion Blur Weiner Weiner 
Spatial Point of Amplitude Amplitude Correction Corrected 
5 Frequency lighl (perfect \ms^ge) w/Blur Filter Factor Amplitude 







0 1 


1 


1 


1 


1 






1 1 


.96 


.92 


1.04 


.957 






2 1 


.85 


.7 


1.172 


.82 




10 


3 1 


.69 


.44 


1.418 


.623 






4 1 


.5 


.2 


1.9 


.38 






5 1 


.31 


.05 


1.944 


.097 






6 1 


.15 


.004 


1.366 


.005 






7 1 


.04 


006 


578 


003 




15 


8 1 


0 


0 


0 


0 






9 1 


.04 


.006 


.578 


.003 


%. i 




10 1 


.15 


.004 


1.366 


.005 


W i 




11 1 


.31 


.05 


1.944 


.097 






12 1 


.5 


.2 


1.9 


.38 


\^ 


20 


13 1 


.69 


.44 


1.418 


.623 






14 1 


.85 


.7 


1.172 


.82 






15 1 


.96 


.92 


1.04 


.957 



As shown by the data of Table 1, motion blur attenuated the high 
25 spatial frequency content of the transformed image. More specifically, the 
values of the high spatial frequencies of five, six, seven, and eight are 
attenuated. Motion blur did not attenuate the values of the lower spatial 
frequencies. 

Various image sharpening kernels can be applied to the image data in 
30 the direction of the motion blur in order to increase the magnitude of the 

amplitude of the high spatial frequency content. The result of the increased 
amplitude of the high spatial frequency content is a sharper image, which is a 
reduction of the blur caused by motion. More specifically, the transitions 
between light and dark areas are accentuated. 
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Table 1 shows the Weiner correction factor that is applied to the signal 
at the specific spatial frequencies in the direction of the motion blur so as to 
reduce motion blur. It should be noted that the Weiner correction factor used 
herein has a minimal impact on the high spatial frequency content. As shown 
5 in Table 1. the frequencies seven and eight are minimally impacted by the 

Weiner correction factor applied herein. This is because the very high spatial 
frequency content of the image data tends to include a lot of noise. Thus, if 
the very high spatial frequency content is amplified, the noise will likely be 
amplified more than the signal. 
1 0 The resulting image after the application of the Weiner correction 

factor is shown in Fig. 8, which is a graph of the point of light of Fig. 5C 
imaged under a motion blur condition, but with the Weiner correction factor 
□ applied. The original motion blurred image of Fig. 5C is shown in dashed 

^ lines in Fig. 8 for reference purposes. The values of the blurred image of Fig. 

Oi 15 5C are compared to the values of the corrected image of Fig. 8 in Table 2 
yj under the portion titled Weiner Corrected Image. It should be noted that the 

Wiener corrected image factor corrected the image for two pixels or 
^ photodetecting elements 138, Fig. 2, of motion blur. The Weiner correction 

m algorithm may be modified to correct for any number of pixels of motion blur, 

fy 20 



25 



30 
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Table 2 





5 


Spatial 
Locgtion 


Point 
of 

Light 


Blur 
Filter 


Motion 
Blurred 
Image 


Weiner 
Corrected 
Image 


-1, 3, -1 
Sharpening 
Kernel 


-0.5, 2, -0.5 
Sharpening 
Kernel 






1 


0 


0 


0 


0.03 


0 


0 






z 


0 


0 


0 


0 


0 


0 






Q 

o 


e\ 
U 


U 


U 


-U.U4 


U 


0 






4 


u 


U 


U 


n no 


U 


0 






c 
O 


U 


U 


U 


U.Ud 


U 


0 






D 


u 


U 


U 


n on 
-U.2U 


-U.bU 


-0.30 






"7 


0 


0 


U.bO 


0.2o 


-0.20 


0.20 






o 
O 


0 


2 


2.00 


2.17 


2.60 


2.30 


tt- 


15 


9 


o 

o 


4 


2.80 


3.37 


4.40 


3.60 






10 


0 


2 


2.00 


2.17 


2.60 


2.30 






11 


0 


0 


0.60 


0.28 


-.020 


0.20 


1 y 




12 


0 


0 


0 


-0.20 


-0.60 


-0.30 






13 


0 


0 


0 


0.60 


0 


0 


O 


20 


14 


0 


0 


0 


0.02 


0 


0 


ru 




15 


0 


0 


0 


-0.04 


0 


0 



As illustrated by Table 2, the Weiner corrected image restores the 
image to a close approximation to the image represented by the blur filter 

25 column. It should be noted that the value of the highest amplitude at the 

spatial location nine ideally would be restored to a value of four, however, the 
Weiner correction factor as described herein restored it to a value of 3.37 
from its motion blurred value of 2.8. It should also be noted that the Weiner 
correction factor illustrated herein has very few negative numbers, which 

30 have very small magnitudes. High magnitude negative numbers tend to 

increase the Gibbs effect, which is detrimental to image quality. The -1, 3, -1 
sharpening kernel did increase the value of the highest spatial frequency. 
However, the value of 4.4 is much greater than the ideal value of 4.0, which 



Docket No. 10992423-1 



increases the noise and is actually detrimental to the image quality. 

As described above, different sharpening algorithms can be applied to 
the image data in the direction of the motion blur in order to reduce the 
motion blur. In Table 2, two alternative examples of sharpening algorithms or 
5 kernels are provided. More specifically, the results of a -1, 3, -1 and a -0.5, 2, 
-0.5 sharpening kernels are shown in Table 2. 

Having described an embodiment of the motion blur detection and 
minimization methods, other embodiments will now be described. 

The method described herein corrects for two pixels of motion blur. It 
10 is to be understood, however, that the method may be modified to correct for 
different amounts of motion blur. In one embodiment, a determination is 
made as to the amount of motion blur in the image. The image is then 
O *deblurred' an amount based on the amount of blur in the image. For 

g example, the method may be able to correct for two or four pixels of motion 

15 blur. If the image is determined to have greater than one and less than three 
yj pixels of motion blur, the image may be corrected for two pixels of motion 

yi 

ffj blur. If the image is determined to have three to five pixels of motion blur, the 

^ image is corrected for four pixels of motion blur. 

Ul The detection of motion blur has been described herein as being 

[ f^ 20 accomplished by analyzing image data representative of an object. Motion 
O blur may also be detected by using a sensor or plurality of sensors mounted 

to the camera 100, Fig. 1 , that detect the amount and direction of movement 
of the camera during the period in which image data is being generated. The 
direction and amount of motion blur then establish the coefficients for the 
25 image sharpening algorithms to reduce the motion blur. 

Referring to Fig. 1 , the detection and minimization of motion blur has 
been described herein as being performed by the processor 132 located 
within the camera 100. In another embodiment, the processor 132 may 
facilitate the transfer of image data to a peripheral processor that may detect 
30 and minimize motion blur. For example, a blurred photograph may be 

captured or othenA^ise scanned so as to generate image data representative 
of the object. 
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Referring again to Fig. 6, the figures of merit have been calculated 
based on image data that is generated by photodetecting elements 1,38, Fig. 
2, that are located next to each other or next to each other in a given color 
plane. One embodiment of the above-described method bases the 
5 calculation of the figures of merit on image data that is generated by 

photodetecting elements 138 that are spaced a distance from each other. 
This will measure the figure of merit at a different spatial frequency. For 
example, the figures of merit may be calculated based on image data 
generated by every other or every third photodetecting element 138. This 
10 embodiment may be used in processing information from digital cameras 
having a great number of photodetecting elements 138. By not processing 
the image data generated by all the photodetecting elements 138, the time 
Q required to determine motion blur is decreased. Several figures of merit at 

g several spatial frequencies can be combined using weighting factors to make 

P 15 the overall figure of merit calculation more immune to image or scene 
yj content. Additionally, several color planes may be weighted in a similar 

pj manner. Likewise, a combination of color planes and spatial frequencies may 

^ be weighted. 

m In another embodiment, small portions of the image are analyzed to 

20 determine if motion blur exists in a small portion of the image. The motion 
O blur is then minimized in that small section. For example, the camera may be 

held stationary relative to a background as image data is being generated. 
An object being imaged, however, may be moving relative to the camera and 
the background as image data is being generated. The method described 
25 above, will reduce the motion blur of the image of the object without creating 
blur in the background. 

The above described method of minimizing motion blur becomes more 
useful as the density of photodetecting elements located on a two- 
dimensional photosensor array increases. The purpose of having an 
30 increased density of photodetecting elements is to provide a more defined or 
sharper image. The susceptibility to motion blur, however, increases as the 
density of photodetecting elements increases. During the period that the 
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photodetecting elements are generating image data, the image of the object 
is more likely to shift between photodetecting elements when the density of 
photodetecting elements is high. This shifting will be detected as motion blur 
and will be minimized accordingly. 

The above described method of minimizing motion blur also becomes 
more useful in generating image data in relatively dark environments. The 
period required by the photodetecting elements to generate image data is 
proportional to the amount of light reflecting from the object being imaged. 
Accordingly, when imaging is performed in a relatively dark environment, the 
period required to generate the image data increases. This increased period 
increases the likelihood that the user of the camera will move the camera 
while image data is being generated, which will cause motion blur. By 
implementing the above-described procedure, the motion blur can be 
minimized. 

Other embodiments for determining and minimizing motion blur may 
be employed using the methods described herein. In one embodiment, 
several color planes are analyzed to determine the motion blur. These 
include the green, red, and blue color planes. In another embodiment, more 
than four directions are analyzed for the presence of motion blur. For 
example, eight directions may be analyzed, which offers a more precise 
determination of the direction of motion blur. In yet another embodiment, the 
image data is analyzed to determine if different amounts of motion blur are 
present. For example, the image data may be analyzed to determine if no 
motion blur exists, two pixel motion blur exists, four pixel motion blur exists 
and so on. 

While an illustrative and presently preferred embodiment of the 
invention has been described in detail herein, it is to be understood that the 
inventive concepts may be otherwise variously embodied and employed and 
that the appended claims are intended to be construed to include such 
variations except insofar as limited by the prior art. 
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